Attentiveness determination device, attentiveness determination system, attentiveness determination method, and computer-readable storage medium

ABSTRACT

An attentiveness determination device includes an image processing unit, a pupil distance calculation unit, a heterophoria detection unit, and an attentiveness determination unit. The image processing unit outputs first reference coordinates, second reference coordinates, first pupil coordinates, and second pupil coordinates. The pupil distance calculation unit calculates at least one position component of the first pupil coordinates with respect to the first reference coordinates and at least one position component of the second pupil coordinates with respect to the second reference coordinates. The heterophoria detection unit outputs a heterophoria detection result indicating a state of a first eyeball and a second eyeball. The attentiveness determination unit determines attentiveness of a person according to the heterophoria detection result.

TECHNICAL FIELD

The present invention relates to an attentiveness determination device,an attentiveness determination system, an attentiveness determinationmethod, and a program.

BACKGROUND ART

There has been proposed an information presentation device that detectsa saccadic eye movement based on face images of a person and determinesan attentiveness level (also referred to simply as “attentiveness”) ofthe person based on the occurrence frequency of the detected saccadiceye movement (see Patent Reference 1, for example).

PRIOR ART REFERENCE Patent Reference

-   Patent Reference 1: Japanese Patent Application Publication No.    11-276461

SUMMARY OF THE INVENTION Problem to be Solved by the Invention

The information presentation device described in the Patent Reference 1detects a saccade that is a high-speed eye movement and determines theattentiveness level of an operator based on the occurrence frequency ofthe detected saccade. However, in order to capture images of ahigh-speed eye movement like the saccade, a camera capable of capturingimages at a high frame rate is necessary. As a result, the cost for theinformation presentation device increases.

An object of the present invention, which is made to resolve theabove-described problem, is to determine the attentiveness of a personby using images captured at a low frame rate.

Means for Solving the Problem

An attentiveness determination device according to an aspect of thepresent invention is an attentiveness determination device to use acaptured image including a first eyeball and a second eyeball of aperson, including:

an image processing unit to set first reference coordinates regardingthe first eyeball and second reference coordinates regarding the secondeyeball in the captured image, calculate first pupil coordinates thatare coordinates of a pupil of the first eyeball in the captured imageand second pupil coordinates that are coordinates of a pupil of thesecond eyeball in the captured image, and output the first referencecoordinates, the second reference coordinates, the first pupilcoordinates, and the second pupil coordinates;

a pupil distance calculation unit to calculate at least one positioncomponent of the first pupil coordinates with respect to the firstreference coordinates and at least one position component of the secondpupil coordinates with respect to the second reference coordinates;

a heterophoria detection unit to output a heterophoria detection resultindicating a state of the first eyeball and the second eyeball by usingthe at least one position component of the first pupil coordinates andthe at least one position component of the second pupil coordinates; andan attentiveness determination unit to determine attentiveness of theperson according to the heterophoria detection result.

An attentiveness determination device according to another aspect of thepresent invention is an attentiveness determination device to use acaptured image including a first eyeball and a second eyeball of aperson, including:

an image processing unit to set first reference coordinates regardingthe first eyeball and second reference coordinates regarding the secondeyeball in the captured image, calculate first pupil coordinates thatare coordinates of a pupil of the first eyeball in the captured imageand second pupil coordinates that are coordinates of a pupil of thesecond eyeball in the captured image, and output the first referencecoordinates, the second reference coordinates, the first pupilcoordinates and the second pupil coordinates;

a pupil distance calculation unit to calculate at least one positioncomponent of the first pupil coordinates with respect to the firstreference coordinates and at least one position component of the secondpupil coordinates with respect to the second reference coordinates;

a pupil distance correction unit to normalize the at least one positioncomponent of the first pupil coordinates and the at least one positioncomponent of the second pupil coordinates and output the normalizedvalues as pupil distance correction values;

a heterophoria detection unit to output a heterophoria detection resultindicating a state of the first eyeball and the second eyeball by usingthe pupil distance correction values; and

an attentiveness determination unit to determine attentiveness of theperson according to the heterophoria detection result.

An attentiveness determination system according to another aspect of thepresent invention includes the attentiveness determination devicedescribed above.

An attentiveness determination method according to another aspect of thepresent invention is an attentiveness determination method ofdetermining attentiveness of a person by using a captured imageincluding a first eyeball and a second eyeball of the person, including:

setting first reference coordinates regarding the first eyeball andsecond reference coordinates regarding the second eyeball in thecaptured image;

calculating first pupil coordinates that are coordinates of a pupil ofthe first eyeball in the captured image and second pupil coordinatesthat are coordinates of a pupil of the second eyeball in the capturedimage;

outputting the first reference coordinates, the second referencecoordinates, the first pupil coordinates and the second pupilcoordinates;

calculating at least one position component of the first pupilcoordinates with respect to the first reference coordinates and at leastone position component of the second pupil coordinates with respect tothe second reference coordinates;

outputting a heterophoria detection result indicating a state of thefirst eyeball and the second eyeball by using the at least one positioncomponent of the first pupil coordinates and the at least one positioncomponent of the second pupil coordinates; and

determining the attentiveness of the person according to theheterophoria detection result.

An attentiveness determination method according to another aspect of thepresent invention is an attentiveness determination method ofdetermining attentiveness of a person by using a captured imageincluding a first eyeball and a second eyeball of the person, including:

setting first reference coordinates regarding the first eyeball andsecond reference coordinates regarding the second eyeball in thecaptured image;

calculating first pupil coordinates that are coordinates of a pupil ofthe first eyeball in the captured image and second pupil coordinatesthat are coordinates of a pupil of the second eyeball in the capturedimage;

outputting the first reference coordinates, the second referencecoordinates, the first pupil coordinates and the second pupilcoordinates;

calculating at least one position component of the first pupilcoordinates with respect to the first reference coordinates and at leastone position component of the second pupil coordinates with respect tothe second reference coordinates;

normalizing the at least one position component of the first pupilcoordinates and the at least one position component of the second pupilcoordinates;

outputting the normalized values as pupil distance correction values;

outputting a heterophoria detection result indicating a state of thefirst eyeball and the second eyeball by using the pupil distancecorrection values; and

determining the attentiveness of the person according to theheterophoria detection result.

A program according to another aspect of the present invention is aprogram that causes a computer to execute an attentiveness determinationmethod of determining attentiveness of a person by using a capturedimage including a first eyeball and a second eyeball of the person, theprogram causing the computer to execute:

setting first reference coordinates regarding the first eyeball andsecond reference coordinates regarding the second eyeball in thecaptured image;

calculating first pupil coordinates that are coordinates of a pupil ofthe first eyeball in the captured image and second pupil coordinatesthat are coordinates of a pupil of the second eyeball in the capturedimage;

outputting the first reference coordinates, the second referencecoordinates, the first pupil coordinates and the second pupilcoordinates;

calculating at least one position component of the first pupilcoordinates with respect to the first reference coordinates and at leastone position component of the second pupil coordinates with respect tothe second reference coordinates;

outputting a heterophoria detection result indicating a state of thefirst eyeball and the second eyeball by using the at least one positioncomponent of the first pupil coordinates and the at least one positioncomponent of the second pupil coordinates; and

determining the attentiveness of the person according to theheterophoria detection result.

A program according to another aspect of the present invention is aprogram that causes a computer to execute an attentiveness determinationmethod of determining attentiveness of a person by using a capturedimage including a first eyeball and a second eyeball of the person, theprogram causing the computer to execute:

setting first reference coordinates regarding the first eyeball andsecond reference coordinates regarding the second eyeball in thecaptured image;

calculating first pupil coordinates that are coordinates of a pupil ofthe first eyeball in the captured image and second pupil coordinatesthat are coordinates of a pupil of the second eyeball in the capturedimage;

outputting the first reference coordinates, the second referencecoordinates, the first pupil coordinates and the second pupilcoordinates;

calculating at least one position component of the first pupilcoordinates with respect to the first reference coordinates and at leastone position component of the second pupil coordinates with respect tothe second reference coordinates;

normalizing the at least one position component of the first pupilcoordinates and the at least one position component of the second pupilcoordinates;

outputting the normalized values as pupil distance correction values;

outputting a heterophoria detection result indicating a state of thefirst eyeball and the second eyeball by using the pupil distancecorrection values; and

determining the attentiveness of the person according to theheterophoria detection result.

Effect of the Invention

According to the present invention, the attentiveness of a person can bedetermined by using images captured at a low frame rate.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a block diagram schematically showing a configuration of anattentiveness determination system according to a first embodiment ofthe present invention.

FIG. 2 is a flowchart showing an example of a process of anattentiveness determination method in the first embodiment.

FIG. 3 is a diagram showing a correspondence relationship betweenpositions of a first eyeball, a second eyeball and a nose of a personand positions of these elements in a captured image.

FIG. 4 is a diagram showing a method of calculating a pupil distance.

FIGS. 5A to 5D are diagrams showing examples of both eyes movement of aperson.

FIGS. 6A to 6D are diagrams showing examples of monocular eyeballmovement, specifically, a heterophoric state, of the person.

FIG. 7A is a diagram showing an example of a hardware configuration ofan attentiveness determination device.

FIG. 7B is a diagram showing another example of the hardwareconfiguration of the attentiveness determination device.

FIG. 8 is a block diagram schematically showing a configuration of anattentiveness determination system according to a second embodiment ofthe present invention.

FIG. 9 is a flowchart showing an example of a process of anattentiveness determination method in the second embodiment.

MODE FOR CARRYING OUT THE INVENTION First Embodiment

FIG. 1 is a block diagram schematically showing a configuration of anattentiveness determination system 101 according to a first embodimentof the present invention.

The attentiveness determination system 101 includes an image capturingunit 10 and an attentiveness determination device 100 that uses an imagecaptured by the image capturing unit 10 and including a first eyeball S1and a second eyeball S2 of a person H.

The attentiveness determination device 100 includes an image processingunit 20, a pupil distance calculation unit 30, a heterophoria detectionunit 40 and an attentiveness determination unit 50. The attentivenessdetermination device 100 determines the attentiveness of the person H byusing the image including the first eyeball S1 and the second eyeballS2. The attentiveness determination device 100 may further include anoutput device 70.

The image captured by the image capturing unit 10 is an image includingthe face of the person H, for example. The image capturing unit 10captures an image including at least the first eyeball S1 and the secondeyeball S2 of the person H. In other words, the image including the faceof the person H captured by the image capturing unit 10 is an imageincluding at least the first eyeball S1 and the second eyeball S2 of theperson H. In this embodiment, the first eyeball S1 is the right eye ofthe person H and the second eyeball S2 is the left eye of the person H.The image captured by the image capturing unit 10 may be either a stillimage or motion video.

Normally, the image capturing unit 10 captures an image including thefirst eyeball S1, the second eyeball S2 and a nose S3 of the person H.In this case, the image including the face of the person H captured bythe image capturing unit 10 is an image including the first eyeball S1,the second eyeball S2 and the nose S3 of the person H.

The first eyeball S1 and the second eyeball S2 are objects that undergoa determination by the attentiveness determination unit 50.

The image captured by the image capturing unit 10 is referred to as a“captured image A1”. The image capturing unit 10 outputs the capturedimage A1. In this embodiment, the image capturing unit 10 captures theimage including the face of the person H periodically or continuouslyand outputs the captured image A1 periodically or continuously.

The image capturing unit 10 may include a memory that stores thecaptured images A1. In this case, the image capturing unit 10 is capableof storing the captured images A1 in the memory and outputting thecaptured images A1 stored in the memory.

The captured images A1 outputted from the image capturing unit 10 areinputted to the image processing unit 20.

FIG. 2 is a flowchart showing an example of a process in theaforementioned attentiveness determination system 101 regarding anattentiveness determination method for determining the attentiveness ofthe person H.

In step ST1, first reference coordinates regarding the first eyeball S1and second reference coordinates regarding the second eyeball S2 are setin the captured image A1.

In step ST2, first pupil coordinates that are coordinates of a pupil 24(referred to also as a first pupil of a right pupil) of the firsteyeball S1 in the captured image A1 and second pupil coordinates thatare coordinates of a pupil 25 (referred to also as a second pupil of aleft pupil) of the second eyeball S2 in the captured image A1 arecalculated.

In step ST3, the first reference coordinates, the second referencecoordinates, the first pupil coordinates and the second pupilcoordinates are outputted.

In step ST4, at least one position component of the first pupilcoordinates with respect to the first reference coordinates and at leastone position component of the second pupil coordinates with respect tothe second reference coordinates are calculated.

In step ST5, a heterophoria detection result indicating a state of thefirst eyeball S1 and the second eyeball S2 is outputted by using the atleast one position component of the first pupil coordinates and the atleast one position component of the second pupil coordinates.

In step ST6, the attentiveness of the person H is determined accordingto the heterophoria detection result.

In step ST7, an attentiveness state E1 is outputted.

The above-described attentiveness determination method will be describedconcretely below.

The image processing unit 20 acquires the captured image A1. The imageprocessing unit 20 generates output coordinates B1 by using the capturedimage A1 and outputs the generated output coordinates B1. The outputcoordinates B1 are data including at least one set of coordinatescalculated by the image processing unit 20. The output coordinates B1includes, for example, at least one set of reference coordinates, thefirst pupil coordinates Re (referred to also as right pupil coordinates)and the second pupil coordinates Le (referred to also as left pupilcoordinates). The output coordinates B1 may further include othercoordinates such as nose coordinates Nt.

FIG. 3 is a diagram showing a correspondence relationship betweenpositions of the first eyeball S1, the second eyeball S2 and the nose S3of the person H and positions of these elements in the captured imageA1.

In the orthogonal coordinate system shown in FIG. 3, an x-axis direction(x-axis) represents a lateral direction in the captured image A1, and ay-axis direction (y-axis) represents a direction orthogonal to thex-axis direction, that is, an up-down direction, in the captured imageA1.

In the step ST1, the image processing unit 20 sets at least one set ofreference coordinates in the captured image (i.e., the captured imageA1). In this embodiment, the reference coordinates regarding the firsteyeball S1 (referred to also as the first reference coordinates) and thereference coordinates regarding the second eyeball S2 (referred to alsoas the second reference coordinates) are set in the captured image A1.

In this embodiment, the image processing unit 20 selects coordinates ofan eye inner corner 26 of the first eyeball S1 as the referencecoordinates regarding the first eyeball S1, and sets the referencecoordinates regarding the first eyeball S1 as first eye inner cornercoordinates Rc. Similarly, the image processing unit 20 selectscoordinates of an eye inner corner 27 of the second eyeball S2 as thereference coordinates regarding the second eyeball S2, and sets thereference coordinates regarding the second eyeball S2 as second eyeinner corner coordinates Lc.

While the first eye inner corner coordinates Rc are used as thereference coordinates regarding the first eyeball S1 in this embodiment,it is also possible to use other coordinates as the referencecoordinates regarding the first eyeball S1. Similarly, while the secondeye inner corner coordinates Lc are used as the reference coordinatesregarding the second eyeball S2, it is also possible to use othercoordinates as the reference coordinates regarding the second eyeballS2. It is also possible to use the same coordinates as the referencecoordinates regarding the first eyeball S1 and the second eyeball S2.

The first eye inner corner coordinates Rc are the coordinates of the eyeinner corner 26 of the first eyeball S1 in the captured image A1, andthe second eye inner corner coordinates Lc are the coordinates of theeye inner corner 27 of the second eyeball S2 in the captured image A1.

The first eye inner corner coordinates Rc are represented as coordinates(Rcx, Rcy), for example, and the second eye inner corner coordinates Lcare represented as coordinates (Lcx, Lcy), for example. Rcx representsthe x coordinate of the eye inner corner 26, that is, the position ofthe eye inner corner 26 on the x-axis. Rcy represents the y coordinateof the eye inner corner 26, that is, the position of the eye innercorner 26 on the y-axis. Lcx represents the x coordinate of the eyeinner corner 27, that is, the position of the eye inner corner 27 on thex-axis. Lcy represents the y coordinate of the eye inner corner 27, thatis, the position of the eye inner corner 27 on the y-axis.

In the step ST2, the image processing unit 20 calculates the first pupilcoordinates Re and the second pupil coordinates Le. The first pupilcoordinates Re are the coordinates of the pupil 24 of the first eyeballS1 in the captured image A1. The second pupil coordinates Le are thecoordinates of the pupil 25 of the second eyeball S2 in the capturedimage A1. The nose coordinates Nt are the coordinates of a nose tip end28 of the nose S3 in the captured image A1. The nose tip end 28 is a tipend part of the nose S3 in the y-axis direction.

The first pupil coordinates Re are represented as coordinates (Rex,Rey), for example, the second pupil coordinates Le are represented ascoordinates (Lex, Ley), for example, and the nose coordinates Nt arerepresented as coordinates (Ntx, Nty), for example. Rex represents the xcoordinate of the pupil 24, that is, the position of the pupil 24 on thex-axis. Rey represents the y coordinate of the pupil 24, that is, theposition of the pupil 24 on the y-axis. Lex represents the x coordinateof the pupil 25, that is, the position of the pupil 25 on the x-axis.Ley represents the y coordinate of the pupil 25, that is, the positionof the pupil 25 on the y-axis. Ntx represents the x coordinate of thenose tip end 28, that is, the position of the nose tip end 28 on thex-axis. Nty represents the y coordinate of the nose tip end 28, that is,the position of the nose tip end 28 on the y-axis.

In the step ST3, the image processing unit 20 outputs at least one setof pupil coordinates and at least one set of reference coordinates asthe output coordinates B1. In this embodiment, the image processing unit20 outputs the first pupil coordinates Re, the second pupil coordinatesLe, the first reference coordinates and the second referencecoordinates.

For example, when the image processing unit 20 sets the first eye innercorner coordinates Rc as the first reference coordinates and the secondeye inner corner coordinates Lc as the second reference coordinates, theimage processing unit 20 outputs the first pupil coordinates Re, thesecond pupil coordinates Le, the first eye inner corner coordinates Rcand the second eye inner corner coordinates Lc as the output coordinatesB1. In this case, the output coordinates B1 are represented as aone-dimensional sequence (i.e., B1=[Rex, Rey, Lex, Ley, Rcx, Rcy, Lcx,Lcy]), for example.

FIG. 4 is a diagram showing a method of calculating the pupil distance.

The output coordinates B1 are inputted to the pupil distance calculationunit 30. By using the output coordinates B1, the pupil distancecalculation unit 30 calculates a pupil distance 31 (referred to also asa first pupil distance), a pupil distance 32 (referred to also as asecond pupil distance), a pupil distance 33 (referred to also as a thirdpupil distance), a pupil distance 34 (referred to also as a fourth pupildistance), a pupil distance 35 (referred to also as a fifth pupildistance) and a pupil distance 36 (referred to also as a sixth pupildistance).

The pupil distance calculation unit 30 calculates the pupil distances31, 32, 33, 34, 35 and 36 periodically. By the calculation, time-seriesdata is obtained by the pupil distance calculation unit 30.

Specifically, the pupil distance calculation unit 30 periodicallycalculates at least one position component of the first pupilcoordinates Re with respect to the first reference coordinates in thecaptured image A1 and at least one position component of the secondpupil coordinates Le with respect to the second reference coordinates inthe captured image A1. In this embodiment, position components of thefirst pupil coordinates Re are the pupil distances 31, 33 and 35, andposition components of the second pupil coordinates Le are the pupildistances 32, 34 and 36.

The pupil distance 31 is the distance from a first reference position tothe pupil 24. In the captured image A1, the pupil distance 31 is thedistance from the first eye inner corner coordinates Rc as the firstreference coordinates to the first pupil coordinates Re. In thisembodiment, the distance from the first eye inner corner coordinates Rcto the first pupil coordinates Re in the captured image A1 is assumed tobe R.

The pupil distance 32 is the distance from a second reference positionto the pupil 25. In the captured image A1, the pupil distance 32 is thedistance from the second eye inner corner coordinates Lc as the secondreference coordinates to the second pupil coordinates Le. In thisembodiment, the distance from the second eye inner corner coordinates Lcto the second pupil coordinates Le in the captured image A1 is assumedto be L.

The pupil distance 33 is the distance from the first reference positionto the pupil 24 in the lateral direction. In the captured image A1, thepupil distance 33 is the distance from the first eye inner cornercoordinates Rc as the first reference coordinates to the first pupilcoordinates Re in the lateral direction in the captured image A1. Inthis embodiment, the distance from the first eye inner cornercoordinates Rc to the first pupil coordinates Re in the lateraldirection in the captured image A1 is assumed to be Rh.

The pupil distance 34 is the distance from the second reference positionto the pupil 25 in the lateral direction. In the captured image A1, thepupil distance 34 is the distance from the second eye inner cornercoordinates Lc as the second reference coordinates to the second pupilcoordinates Le in the lateral direction in the captured image A1. Inthis embodiment, the distance from the second eye inner cornercoordinates Lc to the second pupil coordinates Le in the lateraldirection in the captured image A1 is assumed to be Lh.

The pupil distance 35 is the distance from the first reference positionto the pupil 24 in the up-down direction. In the captured image A1, thepupil distance 35 is the distance from the first eye inner cornercoordinates Rc as the first reference coordinates to the first pupilcoordinates Re in the up-down direction in the captured image A1. Inthis embodiment, the distance from the first eye inner cornercoordinates Rc to the first pupil coordinates Re in the up-downdirection in the captured image A1 is assumed to be Rv.

The pupil distance 36 is the distance from the second reference positionto the pupil 25 in the up-down direction. In the captured image A1, thepupil distance 36 is the distance from the second eye inner cornercoordinates Lc as the second reference coordinates to the second pupilcoordinates Le in the up-down direction in the captured image A1. Inthis embodiment, the distance from the second eye inner cornercoordinates Lc to the second pupil coordinates Le in the up-downdirection in the captured image A1 is assumed to be Lv.

When the image processing unit 20 selects the first eye inner cornercoordinates Rc and the second eye inner corner coordinates Lcrespectively as the first reference coordinates and the second referencecoordinates, the distance Rh is represented as |Rcx−Rex|, the distanceLh is represented as |Lcx−Lex|, the distance Rv is represented as|Rcy−Rey|, and the distance Lv is represented as |Lcy−Ley|.

In this case, the distance R is represented by expression (1) by usingthe distance Rh and the distance Rv, and the distance L is representedby expression (2) by using the distance Lh and the distance Lv.

R=√{square root over (Rh ² +Rv ²)}  (1)

L=√{square root over (Lh ² +Lv ²)}  (2)

In the step ST4, the pupil distance calculation unit 30 outputs thecalculated distances Rh, Lh, Rv, Lv, R and L as pupil distance outputC1. The pupil distance output C1 is represented as a one-dimensionalsequence (i.e., C1=[Rh, Lh, Rv, Lv, R, L]), for example.

The pupil distance output C1 is inputted to the heterophoria detectionunit 40. The heterophoria detection unit 40 calculates fluctuation inthe position of the pupil 24 and fluctuation in the position of thepupil 25 in a predetermined period by using at least one positioncomponent of the first pupil coordinates Re and at least one positioncomponent of the second pupil coordinates Le. Further, the heterophoriadetection unit 40 determines eyeball movement of the person H by using aresult of calculating the fluctuation in the position of the pupil 24and the fluctuation in the position of the pupil 25.

In other words, the heterophoria detection unit 40 determines theeyeball movement of the person H by using fluctuation in components ofthe pupil distance output C1 as time-series data. The components of thepupil distance output C1 are the distances Rh, Lh, Rv, Lv, R and L.

Specifically, by using the fluctuation in components of the pupildistance output C1, the heterophoria detection unit 40 determineswhether the state of both eyes of the person H is both eyes movement(referred to also as a both eyes movement state) or monocular eyeballmovement (e.g., a heterophoric state) of the person H. For example, whenboth of the fluctuation in the position of the pupil 24 and thefluctuation in the position of the pupil 25 are greater than or equal toa threshold value or less than or equal to a threshold value, theheterophoria detection unit 40 determines that the state of both eyes ofthe person H is the both eyes movement state. In contrast, when one ofthe fluctuation in the position of the pupil 24 or the fluctuation inthe position of the pupil 25 is greater than or equal to a thresholdvalue and the other is less than a threshold value, the heterophoriadetection unit 40 determines that one of the first eyeball S1 or thesecond eyeball S2 is in the heterophoric state (i.e., ocular deviation).The heterophoria detection unit 40 may use either one threshold value ortwo or more threshold values.

The fluctuation in a component of the pupil distance output C1 isrepresented as variance, for example.

Since the heterophoria detection unit 40 makes the determination betweenthe both eyes movement state and the heterophoric state, theattentiveness of the person H can be determined according to the resultof the determination between the both eyes movement state and theheterophoric state.

FIG. 5A to FIG. 5D are diagrams showing examples of the both eyesmovement (i.e., the both eyes movement state) of the person H.

FIG. 5A shows an ocular fixation state. The ocular fixation state is astate in which the left and right eyeballs are fixed and a state inwhich the person H is fixing his/her eyes on a visual target.

FIG. 5B is a diagram showing a line-of-sight movement state in thelateral direction.

For example, when the distance Rh increases and the distance Lhdecreases, it is determined that the line of sight of the person H ispointed to the right. In contrast, when the distance Rh decreases andthe distance Lh increases, it is determined that the line of sight ofthe person H is pointed to the left.

FIG. 5C is a diagram showing a line-of-sight movement state in theup-down direction.

For example, when the distance Rv and the distance Lv increase, it isdetermined that the line of sight of the person H is pointed upward. Incontrast, when the distance Rv and the distance Lv increase, it isdetermined that the line of sight of the person H is pointed downward.When the distance R and the distance L increase, it is determined thatthe line of sight of the person H is pointed in an oblique direction.

FIG. 5D is a diagram showing a convergence movement state. Theconvergence movement is a movement of directing both eyes towards thenose. That is, the convergence movement state is a state in which botheyes of the person H are performing the convergence movement. Forexample, when the distance Rh and the distance Lh decrease, the state ofboth eyes of the person H is determined to be the convergence movementstate.

The both eyes movement of the person H is not limited to the examplesshown in FIG. 5A to FIG. 5D. For example, when the distance Rh and thedistance Lh increase, it is determined that both eyes of the person Hare performing a divergence movement. The divergence movement is amovement of directing both eyes towards the ears.

FIG. 6A to FIG. 6D are diagrams showing examples of the monoculareyeball movement, specifically, the heterophoric state, of the person H.

In general, as ocular positions of the human, there are a binocularvision position, a fusion-free position, a physiological rest positionand an absolute rest position. In the binocular vision position,extraocular muscles and intraocular muscles are strained to implementthe binocular vision function. In the fusion-free position, fusionalconvergence for the fusion of images inputted to the left and righteyeballs is removed. The physiological rest position appears in a deepsleep, for example, in which stimuli received by the ocular musclesdecrease to the minimum. The absolute rest position appears after death,for example, in which the ocular muscles are released from all types ofstimuli.

The heterophoria means potentially having the fusion-free positionalthough usually having the binocular vision function by strainingocular muscles, that is, a state of temporarily approaching thefusion-free position when the strain of the ocular muscles becomesinsufficient. When the fusional convergence is gradually lost from thestate in which the binocular vision function works, ocular deviationappears in one eye as the heterophoria, in which the direction of theline of sight varies from person to person.

For example, when a variance value or that is the variance value of thedistance R is less than a threshold value Tr and a variance value ofthat is the variance value of the distance L is greater than or equal toTl, the heterophoria detection unit 40 can determine that the person His (i.e., the first eyeball S1 and the second eyeball S2 are) in one ofthe heterophoric states shown in FIG. 6A to FIG. 6D.

FIG. 6A shows an exophoric state. The exophoric state is a state inwhich one of the pupil of the first eyeball S1 or the pupil of thesecond eyeball S2 is directed towards the ear's side. For example, whenthe distance Rh is constant and the distance Lh increases as shown inFIG. 6A, the first eyeball S1 and the second eyeball S2 are determinedto be in the exophoric state. Similarly, when the distance Rh increasesand the distance Lh is constant, the first eyeball S1 and the secondeyeball S2 are determined to be in the exophoric state.

FIG. 6B shows an esophoric state. The esophoric state is a state inwhich one of the pupil of the first eyeball S1 or the pupil of thesecond eyeball S2 is directed towards the nose's side. For example, whenthe distance Rh is constant and the distance Lh decreases as shown inFIG. 6B, the first eyeball S1 and the second eyeball S2 are determinedto be in the esophoric state. Similarly, when the distance Rh decreasesand the distance Lh is constant, the first eyeball S1 and the secondeyeball S2 are determined to be in the esophoric state.

FIG. 6C shows an anaphoric state. The anaphoric state is a state inwhich one of the pupil of the first eyeball S1 or the pupil of thesecond eyeball S2 is directed upward. For example, when the distance Rvis constant and the distance Lv increases as shown in FIG. 6C, the firsteyeball S1 and the second eyeball S2 are determined to be in theanaphoric state. Similarly, when the distance Rv increases and thedistance Lv is constant, the first eyeball S1 and the second eyeball S2are determined to be in the anaphoric state.

FIG. 6D shows a hypophoric state. The hypophoric state is a state inwhich one of the pupil of the first eyeball S1 or the pupil of thesecond eyeball S2 is directed downward. For example, when the distanceRv is constant and the distance Lv decreases as shown in FIG. 6D, thefirst eyeball S1 and the second eyeball S2 are determined to be in thehypophoric state. Similarly, when the distance Rv decreases and thedistance Lv is constant, the first eyeball S1 and the second eyeball S2are determined to be in the hypophoric state.

The heterophoric state is not limited to the examples shown in FIG. 6Ato FIG. 6D. For example, when one of the exophoric state or theesophoric state and one of the anaphoric state or the hypophoric stateoccur at the same time, the first eyeball S1 and the second eyeball S2are determined to be heterophoric in an oblique direction.

When the heterophoria detection unit 40 determines the state of botheyes of the person H, the heterophoria detection unit 40 calculatesvariance values σrh, σlh, σrv, σlv, or and σl of the components includedin the pupil distance output C1 by using time-series data, for example.In this case, the time-series data is the pupil distance output C1periodically inputted to the heterophoria detection unit 40.

The variance value σrh is the variance value of the distance Rh inputtedto the heterophoria detection unit 40 in a predetermined period. Thevariance value σlh is the variance value of the distance Lh inputted tothe heterophoria detection unit 40 in the predetermined period. Thevariance value σrv is the variance value of the distance Rv inputted tothe heterophoria detection unit 40 in the predetermined period. Thevariance value σlv is the variance value of the distance Lv inputted tothe heterophoria detection unit 40 in the predetermined period. Thevariance value or is the variance value of the distance R inputted tothe heterophoria detection unit 40 in the predetermined period. Thevariance value σl is the variance value of the distance L inputted tothe heterophoria detection unit 40 in the predetermined period.

The heterophoria detection unit 40 compares the variance value of eachcomponent with a predetermined threshold value (referred to also as a“fluctuation threshold value”) corresponding to the variance value.

The threshold value corresponding to the variance value σrh is athreshold value Trh. The threshold value corresponding to the variancevalue σlh is a threshold value Tlh. The threshold value corresponding tothe variance value σrv is a threshold value Try. The threshold valuecorresponding to the variance value σlv is a threshold value Tlv. Thethreshold value corresponding to the variance value or is a thresholdvalue Tr. The threshold value corresponding to the variance value σl isa threshold value Tl.

Each of the threshold values Trh, Tlh, Try, Tlv, Tr and Tl is apredetermined value. As each threshold value, it is possible to use thevariance value of each component included in the pupil distance outputC1 in a predetermined period, or a value obtained by weighting avariance value acquired from time-series data in the ocular fixationstate with a weight.

For example, the heterophoria detection unit 40 determines whether ornot data regarding the first eyeball S1 satisfies a first condition(i.e., σrh<Trh, σrv<Try and or <Tr) and determines whether or not dataregarding the second eyeball S2 satisfies a second condition (i.e.,σlh<Tlh, σlv<Tlv and σl<Tl).

When the first condition (i.e., σrh<Trh, σrv<Try and or <Tr) issatisfied, the heterophoria detection unit 40 determines that the firsteyeball S1 of the person H is in the ocular fixation state.

In contrast, when the first condition is not satisfied, the heterophoriadetection unit 40 determines that the first eyeball S1 of the person His not in the ocular fixation state. That is, the heterophoria detectionunit 40 determines that the first eyeball S1 is moving in somedirection.

When the second condition (i.e., σlh<Tlh, σlv<Tlv and σl<Tl) issatisfied, the heterophoria detection unit 40 determines that the secondeyeball S2 of the person H is in the ocular fixation state.

In contrast, when the second condition is not satisfied, theheterophoria detection unit 40 determines that the first eyeball S1 ofthe person H is not in the ocular fixation state. That is, theheterophoria detection unit 40 determines that the second eyeball S2 ismoving in some direction.

When the data regarding the first eyeball S1 and the second eyeball S2do not satisfy both of the first condition and the second condition, theheterophoria detection unit 40 determines that the first eyeball S1 andthe second eyeball S2 are performing both eyes movement such as sightline movement in the up-down direction, sight line movement in thelateral direction, convergence movement or divergence movement. In otherwords, when the data regarding the first eyeball S1 and the secondeyeball S2 do not satisfy both of the first condition and the secondcondition, the heterophoria detection unit 40 determines that the stateof the first eyeball S1 and the second eyeball S2 is the both eyesmovement state.

When the data regarding the first eyeball S1 and the second eyeball S2satisfy only one of the first condition or the second condition, theheterophoria detection unit 40 determines that the ocular deviation ofone eye of the person H occurs. In this case, the heterophoria detectionunit 40 determines that the state of the person H is the heterophoricstate. In other words, the heterophoria detection unit 40 determinesthat one of the first eyeball S1 or the second eyeball S2 is in theheterophoric state.

By determining behavior of both eyes of the person H, it is possible todetermine whether the first eyeball S1 and the second eyeball S2 are inthe both eyes movement state or one of the first eyeball S1 or thesecond eyeball S2 is in the heterophoric state.

The heterophoria detection unit 40 has only to determine whether thefirst eyeball S1 and the second eyeball S2 are in the both eyes movementstate or one of the first eyeball S1 or the second eyeball S2 is in theheterophoric state. The above-described determination method of theheterophoria detection unit 40 is just an example and variousdetermination conditions may be combined together.

In the step ST5, the heterophoria detection unit 40 outputs the resultof the determination as a heterophoria detection result D1. Theheterophoria detection result D1 indicates the state of the firsteyeball S1 and the second eyeball S2. For example, the heterophoriadetection result D1 indicates whether the first eyeball S1 and thesecond eyeball S2 are in the both eyes movement state or one of thefirst eyeball S1 or the second eyeball S2 is in the heterophoric state.

The heterophoria detection result D1 is inputted to the attentivenessdetermination unit 50. In the step ST6, the attentiveness determinationunit 50 determines the attentiveness of the person H according to theheterophoria detection result D1 and generates the result of thedetermination as the attentiveness state E1.

The attentiveness state E1 is, for example, an attentivenessdeteriorating state or an attentiveness maintaining state. In this case,the attentiveness deteriorating state is a state in which theattentiveness of the person H is low, and the attentiveness maintainingstate is a state in which the attentiveness of the person H is high.

For example, when the heterophoria detection result D1 indicates theboth eyes movement state, the attentiveness determination unit 50determines that the person H is in the attentiveness maintaining stateand generates the attentiveness state E1 as a signal indicating theattentiveness maintaining state. In contrast, when the heterophoriadetection result D1 indicates the heterophoric state, the attentivenessdetermination unit 50 determines that the person H is in theattentiveness deteriorating state and generates the attentiveness stateE1 as a signal indicating the attentiveness deteriorating state.

In the step ST7, the attentiveness determination unit 50 outputs theattentiveness state E1. The attentiveness state E1 is inputted to theoutput device 70 such as a monitor, a head-up display, a speaker or avibrator, for example. Depending on the attentiveness state E1, theoutput device 70 outputs at least one of an image (e.g., a still imageor motion video), audio, or vibration, for example. When theattentiveness state E1 is not the attentiveness deteriorating state, theoutput device 70 does not need to output anything.

FIG. 7A is a diagram showing an example of a hardware configuration ofthe attentiveness determination device 100.

FIG. 7B is a diagram showing another example of the hardwareconfiguration of the attentiveness determination device 100.

The attentiveness determination device 100 is formed of at least oneprocessor 108 a and at least one memory 108 b, for example. Theprocessor 108 a is, for example, a central processing unit (CPU) thatexecutes a program stored in the memory 108 b. In this case, thefunctions of the attentiveness determination device 100 are implementedby software, firmware, or a combination of software and firmware. Thesoftware and the firmware can be stored in the memory 108 b as programs.With this configuration, a program for implementing the functions of theattentiveness determination device 100 (e.g., the attentivenessdetermination method described in this embodiment) is executed by acomputer.

The memory 108 b as a computer-readable storage medium is, for example,a volatile memory, a nonvolatile memory, or a combination of a volatilememory and a nonvolatile memory such as a RAM (Random Access Memory) anda ROM (Read Only Memory).

The attentiveness determination device 100 may also be formed ofprocessing circuitry 108 c as dedicated hardware such as a singlecircuit or a combined circuit. In this case, the functions of theattentiveness determination device 100 are implemented by the processingcircuitry 108 c.

As described above, the attentiveness determination device 100 in thisfirst embodiment detects whether or not the ocular deviation occurs inthe person H. Therefore, the attentiveness determination device 100 iscapable of determining the attentiveness by using images captured at alow frame rate. Accordingly, the attentiveness determination device 100does not need high processing power of the CPU in comparison with thedevice that detects the saccade that is a high-speed eye movement. As aresult, the production cost of the attentiveness determination device100 can be reduced.

The attentiveness determination system 101 according to the firstembodiment includes the attentiveness determination device 100.Accordingly, the attentiveness determination system 101 has advantagesthe same as the aforementioned advantages of the attentivenessdetermination device 100.

Second Embodiment

FIG. 8 is a block diagram schematically showing a configuration of anattentiveness determination system 201 according to a second embodimentof the present invention.

FIG. 9 is a flowchart showing an example of a process in theaforementioned attentiveness determination system 201 regarding anattentiveness determination method for determining the attentiveness ofa person H.

In step ST1, the first reference coordinates regarding the first eyeballS1 and the second reference coordinates regarding the second eyeball S2are set in the captured image A1.

In step ST2, the first pupil coordinates that are the coordinates of thepupil of the first eyeball S1 in the captured image A1 and the secondpupil coordinates that are the coordinates of the pupil of the secondeyeball S2 in the captured image A1 are calculated.

In step ST3, the first reference coordinates, the second referencecoordinates, the first pupil coordinates and the second pupilcoordinates are outputted.

In step ST4, at least one position component of the first pupilcoordinates with respect to the first reference coordinates and at leastone position component of the second pupil coordinates with respect tothe second reference coordinates are calculated.

In step ST5, the at least one position component of the first pupilcoordinates and the at least one position component of the second pupilcoordinates are normalized.

In step ST6, the normalized values are outputted as pupil distancecorrection values.

In step ST7, the heterophoria detection result indicating the state ofthe first eyeball S1 and the second eyeball S2 is outputted by using thepupil distance correction values.

In step ST8, the attentiveness of the person H is determined accordingto the heterophoria detection result.

In step ST9, the attentiveness state E1 is outputted.

In the second embodiment, the description will be given mainly ofconfigurations and operations different from those in the firstembodiment.

The attentiveness determination system 201 according to the secondembodiment includes an attentiveness determination device 200 instead ofthe attentiveness determination device 100. In the second embodiment,the attentiveness determination device 200 uses images including thefirst eyeball S1, the second eyeball S2 and the nose S3 of the person Hcaptured by the image capturing unit 10.

The attentiveness determination device 200 includes the image capturingunit 10, the image processing unit 20, the pupil distance calculationunit 30, the heterophoria detection unit 40, the attentivenessdetermination unit 50 and a pupil distance correction unit 60. Putanother way, the attentiveness determination device 200 according to thesecond embodiment includes the pupil distance correction unit 60 inaddition to the image capturing unit 10, the image processing unit 20,the pupil distance calculation unit 30, the heterophoria detection unit40 and the attentiveness determination unit 50 described in the firstembodiment. The attentiveness determination device 200 may furtherinclude the output device 70.

The attentiveness determination device 200 determines the attentivenessof the person H by using the images including the first eyeball S1, thesecond eyeball S2 and the nose S3.

The hardware configuration of the attentiveness determination device 200may be the same as the hardware configuration described in the firstembodiment. In this case, the hardware configuration of theattentiveness determination device 200 is the hardware configurationshown in FIG. 7A or FIG. 7B.

In the step ST2, the image processing unit 20 further calculates thefirst eye inner corner coordinates Rc of the first eyeball S1, thesecond eye inner corner coordinates Lc of the second eyeball S2 and thenose coordinates Nt in addition to the first pupil coordinates Re andthe second pupil coordinates Le.

In the step ST3, the image processing unit 20 outputs at least one setof pupil coordinates, the nose coordinates Nt and at least one set ofreference coordinates as output coordinates B2. In this embodiment, theimage processing unit 20 outputs the first pupil coordinates Re, thesecond pupil coordinates Le, the nose coordinates Nt, the firstreference coordinates and the second reference coordinates as the outputcoordinates B2.

For example, when the image processing unit 20 sets the first eye innercorner coordinates Rc as the first reference coordinates and the secondeye inner corner coordinates Lc as the second reference coordinatessimilarly to the first embodiment, the image processing unit 20 outputsthe first pupil coordinates Re, the second pupil coordinates Le, thenose coordinates Nt, the first eye inner corner coordinates Rc and thesecond eye inner corner coordinates Lc as the output coordinates B2. Inthis case, the output coordinates B2 are represented as aone-dimensional sequence (i.e., B2=[Rex, Rey, Lex, Ley, Rcx, Rcy, Lcx,Lcy, Ntx, Nty]), for example.

The output coordinates B2 are inputted to the pupil distance calculationunit 30. In the step ST4, by using the output coordinates B2, the pupildistance calculation unit 30 periodically calculates an eye inner cornerdistance 37 and a nose bridge distance 38 in addition to the pupildistances 31, 32, 33, 34, 35 and 36. By the calculation, time-seriesdata is obtained by the pupil distance calculation unit 30.

The eye inner corner distance 37 is the distance between the firsteyeball S1 and the second eyeball S2. Specifically, the eye inner cornerdistance 37 is the distance between the eye inner corner 26 of the firsteyeball S1 and the eye inner corner 27 of the second eyeball S2. In thecaptured image A1, the eye inner corner distance 37 is the distance fromthe first eye inner corner coordinates Rc to the second eye inner cornercoordinates Lc. In this embodiment, the distance from the first eyeinner corner coordinates Rc to the second eye inner corner coordinatesLc in the captured image A1 is assumed to be D.

The nose bridge distance 38 is the distance between a midpoint P1 of theeye inner corner distance 37 and the nose tip end 28. In the capturedimage A1, the nose bridge distance 38 is the distance from the midpointP1 to the nose coordinates Nt. In this embodiment, the distance from themidpoint P1 to the nose coordinates Nt in the captured image A1 isassumed to be N.

The distance D is represented by the following expression (3):

D=√{square root over ((Lcs−Rcx)²+(Lcy+Rcy)²)}  (3)

The distance N is represented by the following expression (4):

N=√{square root over (((Lcx+Rcx)/2−Ntx)²÷((Lcy+Rcy)/2−Nty)²)}  (4)

The pupil distance calculation unit 30 outputs the calculated distancesRh, Lh, Rv, Lv, R, L, D and N as pupil distance output C2. The pupildistance output C2 is represented as a one-dimensional sequence (i.e.,C2=[Rh, Lh, Rv, Lv, R, L, D, N]), for example.

Further, the pupil distance calculation unit 30 may output thecalculated distances D and N as reference value output G. The referencevalue output G is represented as a one-dimensional sequence (i.e., G=[D,N]), for example.

The pupil distance output C2 and the reference value output G areinputted to the pupil distance correction unit 60. In the step ST5, thepupil distance correction unit 60 normalizes the pupil distance, thatis, at least one position component of the first pupil coordinates Reand at least one position component of the second pupil coordinates Le,by using at least one arbitrary value.

In this embodiment, the pupil distance correction unit 60 normalizes thepupil distance, that is, at least one position component of the firstpupil coordinates Re and at least one position component of the secondpupil coordinates Le, by using the eye inner corner distance 37 (i.e.,the distance D) or the nose bridge distance 38 (i.e., the distance N).

Specifically, the pupil distance correction unit 60 normalizes the pupildistance 33 and the pupil distance 34 by using the eye inner cornerdistance 37. For example, the pupil distance 33 is normalized by Rh/D,and the pupil distance 34 is normalized by Lh/D.

The pupil distance correction unit 60 normalizes the pupil distance 35and the pupil distance 36 by using the nose bridge distance 38. Forexample, the pupil distance 35 is normalized by Rv/N, and the pupildistance 36 is normalized by Lv/N.

The pupil distance correction unit 60 updates the pupil distance 31 byusing the normalized pupil distance 35. For example, the updated pupildistance 31 is represented by the following expression (5):

R=√{square root over ((Rh/D)²+(Rv/N)²)}  (5)

The pupil distance correction unit 60 updates the pupil distance 32 byusing the normalized pupil distance 36. For example, the updated pupildistance 32 is represented by the following expression (6):

L=√{square root over ((Lh/D)²+(Lv/N)²)}  (6)

In the step ST6, the pupil distance correction unit 60 outputs thenormalized values (i.e., normalized position components) as pupildistance correction values F. The pupil distance correction values F arerepresented as a one-dimensional sequence (i.e., F=[Rh/D, Lh/D, Rv/N,Lv/N, R, L]), for example.

The pupil distance correction values F are inputted to the heterophoriadetection unit 40. In the step ST7, the heterophoria detection unit 40outputs the heterophoria detection result indicating the state of thefirst eyeball S1 and the second eyeball S2 by using the pupil distancecorrection values F. Specifically, the heterophoria detection unit 40determines the eyeball movement of the person H, that is, the state ofboth eyes of the person H, by using fluctuation in the pupil distancecorrection values F. More specifically, the heterophoria detection unit40 determines whether the state of both eyes of the person H is the botheyes movement or the monocular eyeball movement of the person H by usingthe fluctuation in the pupil distance correction values F. Theheterophoria detection unit 40 outputs the result of the determinationas the heterophoria detection result.

When the heterophoria detection unit 40 determines the state of botheyes of the person H, the heterophoria detection unit 40 calculatesvariance values of the components included in the pupil distancecorrection values F by using time-series data, for example. In thiscase, the time-series data is the pupil distance correction values Fperiodically inputted to the heterophoria detection unit 40.

The heterophoria detection unit 40 compares the variance value of eachcomponent with a predetermined threshold value (referred to also as a“fluctuation threshold value”) corresponding to the variance value.

As described in the first embodiment, the heterophoria detection unit 40determines whether or not data regarding the first eyeball S1 satisfiesthe first condition, determines whether or not data regarding the secondeyeball S2 satisfies the second condition, and outputs the result of thedetermination as the heterophoria detection result D1.

The heterophoria detection result D1 is inputted to the attentivenessdetermination unit 50. Further, the reference value output G, that is,time-series data of the eye inner corner distance 37 and the nose bridgedistance 38, is inputted to the attentiveness determination unit 50. Inthe example shown in FIG. 8, the reference value output G is inputtedfrom the pupil distance calculation unit 30 to the attentivenessdetermination unit 50. However, the reference value output G may also beinputted from a component other than the pupil distance calculation unit30 (e.g., the heterophoria detection unit 40 or the pupil distancecorrection unit 60) to the attentiveness determination unit 50.

In the step ST8, the attentiveness determination unit 50 calculates avariance value od of the eye inner corner distance 37 by using thetime-series data of the eye inner corner distance 37 and calculates avariance value on by using the time-series data of the nose bridgedistance 38. The attentiveness determination unit 50 compares thecalculated variance values with a fluctuation threshold value.

Specifically, the attentiveness determination unit 50 compares thecalculated variance value od with a fluctuation threshold value Td andcompares the calculated variance value on with a fluctuation thresholdvalue Tn.

The fluctuation threshold value Td is a predetermined value. Forexample, as the fluctuation threshold value Td, it is possible to usethe variance value of the eye inner corner distance 37 in apredetermined period, or a value obtained by weighting a variance valueacquired from time-series data in the ocular fixation state with aweight. Similarly, the fluctuation threshold value Tn is a predeterminedvalue. For example, as the fluctuation threshold value Tn, it ispossible to use the variance value of the nose bridge distance 38 in apredetermined period, or a value obtained by weighting a variance valueacquired from time-series data in the ocular fixation state with aweight.

When the time-series data (specifically, the variance value od regardingthe eye inner corner distance 37 and the variance value on regarding thenose bridge distance 38) satisfies an attentiveness condition (i.e.,σd<Td and σn<Tn), the change in the face direction of the person H issmall. Thus, when the time-series data satisfies the attentivenesscondition (i.e., σd<Td and σn<Tn), the attentiveness determination unit50 determines that the person H is in the attentiveness deterioratingstate and generates the attentiveness state E1 indicating theattentiveness deteriorating state.

In contrast, when the variance value σd is greater than or equal to thefluctuation threshold value Td, the face direction of the person H ismoving widely in the Pitch direction (y-axis direction in FIG. 3). Whenthe variance value on is greater than or equal to the fluctuationthreshold value Tn, the face direction of the person H is moving widelyin the Yaw direction (x-axis direction in FIG. 3).

That is, when the time-series data (specifically, the variance value σdregarding the eye inner corner distance 37 and the variance value onregarding the nose bridge distance 38) does not satisfy theattentiveness condition (i.e., σd<Td and σn<Tn), the face direction ofthe person H is moving widely in the Pitch direction and the Yawdirection. In this case, the attentiveness determination unit 50determines that the person H is sufficiently viewing the surroundings.Therefore, when the time-series data does not satisfy the attentivenesscondition, the attentiveness determination unit 50 determines that theperson H is in the attentiveness maintaining state and generates theattentiveness state E1 indicating the attentiveness maintaining state.

In the step ST9, the attentiveness determination unit 50 outputs theattentiveness state E1.

The attentiveness determination device 200 in the second embodiment hasadvantages the same as the advantages of the attentiveness determinationdevice 100 in the first embodiment.

Further, the attentiveness determination device 200 according to thesecond embodiment determines the attentiveness of the person H by usingthe eye inner corner distance 37 and the nose bridge distance 38. Theeye inner corner distance 37 and the nose bridge distance 38 can beregarded as fixed indices of the person H. Therefore, by normalizing thepupil distances (e.g., the pupil distances 33, 34, 35 and 36) by usingthe eye inner corner distance 37 or the nose bridge distance 38,influence of fluctuation in the pupil distances due to a minute changein the face direction of the person H can be reduced. As a result,accuracy of the analysis of the time-series data of the pupil distances(e.g., the pupil distances 33, 34, 35 and 36) can be increased.

The attentiveness determination system 201 including the attentivenessdetermination device 200 has advantages the same as the aforementionedadvantages of the attentiveness determination device 200.

Features in the embodiments described above can be appropriatelycombined with each other.

DESCRIPTION OF REFERENCE CHARACTERS

10: image capturing unit, 20: image processing unit, 30: pupil distancecalculation unit, 40: heterophoria detection unit, 50: attentivenessdetermination unit, 60: pupil distance correction unit, 70: outputdevice, 100, 200: attentiveness determination device, 101, 201:attentiveness determination system.

1. An attentiveness determination device to use a captured imageincluding a first eyeball and a second eyeball of a person, comprising:a processor to execute a program; and a memory to store the programwhich, when executed by the processor, performs processes of, settingfirst reference coordinates regarding the first eyeball and secondreference coordinates regarding the second eyeball in the capturedimage, calculating first pupil coordinates that are coordinates of apupil of the first eyeball in the captured image and second pupilcoordinates that are coordinates of a pupil of the second eyeball in thecaptured image, and outputting the first reference coordinates, thesecond reference coordinates, the first pupil coordinates, and thesecond pupil coordinates; calculating at least one position component ofthe first pupil coordinates with respect to the first referencecoordinates and at least one position component of the second pupilcoordinates with respect to the second reference coordinates; outputtinga heterophoria detection result indicating a state of the first eyeballand the second eyeball by using the at least one position component ofthe first pupil coordinates and the at least one position component ofthe second pupil coordinates; and determining attentiveness of theperson according to the heterophoria detection result.
 2. Theattentiveness determination device according to claim 1, wherein the atleast one position component of the first pupil coordinates includes adistance from the first reference coordinates to the first pupilcoordinates, and the at least one position component of the second pupilcoordinates includes a distance from the second reference coordinates tothe second pupil coordinates.
 3. The attentiveness determination deviceaccording to claim 1, wherein the heterophoria detection unit calculatesfluctuation in a position of the pupil of the first eyeball andfluctuation in a position of the pupil of the second eyeball by usingthe at least one position component of the first pupil coordinates andthe at least one position component of the second pupil coordinates, anddetermines the state of the first eyeball and the second eyeball byusing a result of calculating the fluctuation in the position of thepupil of the first eyeball and the fluctuation in the position of thepupil of the second eyeball.
 4. The attentiveness determination deviceaccording to claim 1, wherein the heterophoria detection unit determinesthat the first eyeball and the second eyeball are in both eyes movementwhen both of fluctuation in a position of the pupil of the first eyeballand fluctuation in a position of the pupil of the second eyeball aregreater than or equal to a threshold value or less than or equal to thethreshold value, and the heterophoria detection unit determines that oneof the first eyeball or the second eyeball is in a heterophoric statewhen one of fluctuation in a position of the pupil of the first eyeballor fluctuation in a position of the pupil of the second eyeball isgreater than or equal to the threshold value and the other is less thanthe threshold value.
 5. The attentiveness determination device accordingto claim 1, wherein the heterophoria detection result indicates whetherthe first eyeball and the second eyeball are in a both eyes movementstate or one of the first eyeball or the second eyeball is in aheterophoric state.
 6. The attentiveness determination device accordingto claim 5, wherein the attentiveness determination unit determines thatthe person is in an attentiveness maintaining state when theheterophoria detection result indicates the both eyes movement state,and the attentiveness determination unit determines that the person isin an attentiveness deteriorating state when the heterophoria detectionresult indicates the heterophoric state.
 7. The attentivenessdetermination device according to claim 6, wherein the attentivenessdetermination unit outputs a signal indicating the attentivenessdeteriorating state or the attentiveness maintaining state of the personaccording to the heterophoria detection result.
 8. An attentivenessdetermination device to use a captured image including a first eyeballand a second eyeball of a person, comprising: a processor to execute aprogram; and a memory to store the program which, when executed by theprocessor, performs processes of, setting first reference coordinatesregarding the first eyeball and second reference coordinates regardingthe second eyeball in the captured image, calculating first pupilcoordinates that are coordinates of a pupil of the first eyeball in thecaptured image and second pupil coordinates that are coordinates of apupil of the second eyeball in the captured image, and outputting thefirst reference coordinates, the second reference coordinates, the firstpupil coordinates, and the second pupil coordinates; calculating atleast one position component of the first pupil coordinates with respectto the first reference coordinates and at least one position componentof the second pupil coordinates with respect to the second referencecoordinates; normalizing the at least one position component of thefirst pupil coordinates and the at least one position component of thesecond pupil coordinates and outputting the normalized values as pupildistance correction values; outputting a heterophoria detection resultindicating a state of the first eyeball and the second eyeball by usingthe pupil distance correction values; and determining attentiveness ofthe person according to the heterophoria detection result.
 9. Anattentiveness determination system comprising: the attentivenessdetermination device according to claim 1; and an image capturing unitto capture the captured image including the first eyeball and the secondeyeball of the person.
 10. An attentiveness determination method ofdetermining attentiveness of a person by using a captured imageincluding a first eyeball and a second eyeball of the person, theattentiveness determination method comprising: setting first referencecoordinates regarding the first eyeball and second reference coordinatesregarding the second eyeball in the captured image; calculating firstpupil coordinates that are coordinates of a pupil of the first eyeballin the captured image and second pupil coordinates that are coordinatesof a pupil of the second eyeball in the captured image; outputting thefirst reference coordinates, the second reference coordinates, the firstpupil coordinates, and the second pupil coordinates; calculating atleast one position component of the first pupil coordinates with respectto the first reference coordinates and at least one position componentof the second pupil coordinates with respect to the second referencecoordinates; outputting a heterophoria detection result indicating astate of the first eyeball and the second eyeball by using the at leastone position component of the first pupil coordinates and the at leastone position component of the second pupil coordinates; and determiningthe attentiveness of the person according to the heterophoria detectionresult.
 11. An attentiveness determination method of determiningattentiveness of a person by using a captured image including a firsteyeball and a second eyeball of the person, the attentivenessdetermination method comprising: setting first reference coordinatesregarding the first eyeball and second reference coordinates regardingthe second eyeball in the captured image; calculating first pupilcoordinates that are coordinates of a pupil of the first eyeball in thecaptured image and second pupil coordinates that are coordinates of apupil of the second eyeball in the captured image; outputting the firstreference coordinates, the second reference coordinates, the first pupilcoordinates, and the second pupil coordinates; calculating at least oneposition component of the first pupil coordinates with respect to thefirst reference coordinates and at least one position component of thesecond pupil coordinates with respect to the second referencecoordinates; normalizing the at least one position component of thefirst pupil coordinates and the at least one position component of thesecond pupil coordinates; outputting the normalized values as pupildistance correction values; outputting a heterophoria detection resultindicating a state of the first eyeball and the second eyeball by usingthe pupil distance correction values; and determining the attentivenessof the person according to the heterophoria detection result.
 12. Acomputer-readable storage medium storing a program that causes acomputer to execute an attentiveness determination method of determiningattentiveness of a person by using a captured image including a firsteyeball and a second eyeball of the person, the program causing thecomputer to execute: setting first reference coordinates regarding thefirst eyeball and second reference coordinates regarding the secondeyeball in the captured image; calculating first pupil coordinates thatare coordinates of a pupil of the first eyeball in the captured imageand second pupil coordinates that are coordinates of a pupil of thesecond eyeball in the captured image; outputting the first referencecoordinates, the second reference coordinates, the first pupilcoordinates, and the second pupil coordinates; calculating at least oneposition component of the first pupil coordinates with respect to thefirst reference coordinates and at least one position component of thesecond pupil coordinates with respect to the second referencecoordinates; outputting a heterophoria detection result indicating astate of the first eyeball and the second eyeball by using the at leastone position component of the first pupil coordinates and the at leastone position component of the second pupil coordinates; and determiningthe attentiveness of the person according to the heterophoria detectionresult.
 13. A computer-readable storage medium storing a program thatcauses a computer to execute an attentiveness determination method ofdetermining attentiveness of a person by using a captured imageincluding a first eyeball and a second eyeball of the person, theprogram causing the computer to execute: setting first referencecoordinates regarding the first eyeball and second reference coordinatesregarding the second eyeball in the captured image; calculating firstpupil coordinates that are coordinates of a pupil of the first eyeballin the captured image and second pupil coordinates that are coordinatesof a pupil of the second eyeball in the captured image; outputting thefirst reference coordinates, the second reference coordinates, the firstpupil coordinates, and the second pupil coordinates; calculating atleast one position component of the first pupil coordinates with respectto the first reference coordinates and at least one position componentof the second pupil coordinates with respect to the second referencecoordinates; normalizing the at least one position component of thefirst pupil coordinates and the at least one position component of thesecond pupil coordinates; outputting the normalized values as pupildistance correction values; outputting a heterophoria detection resultindicating a state of the first eyeball and the second eyeball by usingthe pupil distance correction values; and determining the attentivenessof the person according to the heterophoria detection result.
 14. Anattentiveness determination system comprising: the attentivenessdetermination device according to claim 8; and an image capturing unitto capture the captured image including the first eyeball and the secondeyeball of the person.